Remote copy system maintaining consistency

ABSTRACT

To provide a remote copy system including: a plurality of primary storage systems and a plurality of secondary storage systems; each of the secondary storage systems including a secondary logical volume that stores a copy of the data stored in a primary logical volume of the primary storage system; a pre-update data storage part that stores pre-update data that is stored before data to be stored in the secondary logical volume is updated and time stamps of the data to be stored in the secondary logical volume; pre-update data prior to a recovery time set based on the time stamps stored in the pre-update data storage part is obtained from the pre-update data storage part; and the pre-update data is written to the secondary logical volume in an inverse order of the time stamp with the latest time stamp first, to recover data of the secondary logical volume.

CLAIM OF PRIORITY

The present application claims priority from Japanese applicationP2004-200226 filed on Jul. 17, 2004, the content of which is herebyincorporated by reference into this application.

BACKGROUND

This invention relates to a remote copy system in which a plurality ofstorage systems hold copies of data used by a computer, and moreparticularly to a technique of rolling back a plurality of storagesystems in synchronization with each other.

Computer systems available in recent years use an increased amount ofdata, and have the data updated more frequently. It is a major challengein the storage field how such data is backed up and how rapidly a systemcan be recovered to a normal operation state upon the occurrence of afault. As a measure to this end, there is disclosed a remote copytechnique in which a plurality of storage subsystems (external storagesystems) equipped with magnetic disk arrays are placed in remotelocations and connected to each other via a communication path, and dataupdated in one storage subsystem is automatically copied to anotherstorage subsystem without the intermediation of a host computer (referto JP 2004-13367 A).

In addition, there is proposed a system using a write time as atechnique of ensuring consistency of data between a primary storage anda secondary storage. More specifically, a primary storage systemreceiving write data from a primary host notifies the primary host thatthe primary storage system has received write data immediately afterreceiving the write data. After that, the primary host reads a copy ofthe write data from the primary storage system. Each piece of write datais appended with a write time that is the time when a correspondingwrite request is issued. When the primary host reads the write data, thewrite time is sent to the primary host together. Further, the primaryhost transfers the write data and the write time to a secondary host.

The secondary host that has received the write data and the write timewrites information including the write time into a control volume of asecondary storage system. The secondary host further writes the writedata into the secondary storage system in the order of the write time byreferencing the write time appended to each piece of write data. Bywriting the write data into the secondary storage system in the order ofthe write time, consistent data can be held in the secondary storagesystem at all times (refer to EP 0671686).

SUMMARY

In the above-mentioned related art, a host needs to be operatingconstantly in order to keep performing processes of transferring andmirroring data without intermission. In addition, programs for executingthe above processes need to be running constantly, thereby imposing aprocess load on the host at all times.

Also, in the case where a plurality of storage systems are connected toa plurality of other storage systems via a plurality of paths, sinceasynchronous processes of copying data between the storage systems areperformed at arbitrary timings, time to update data are differentdepending on the paths between the storage systems and the other storagesystems. Accordingly, upon failover, data stored in the storage systemsmay exhibit inconsistency between the paths. This may hinder a secondarysite from providing a service.

It is therefore an object of this invention to implement failoversynchronized between a plurality of paths in the case where theplurality of paths are set as paths for remotely copying a storagesystem to another storage system.

According to an embodiment of the present invention, there is provided aremote copy system for copying data between a plurality of storagesystems, including: a plurality of primary storage systems to and fromwhich data is inputted and outputted; and a plurality of secondarystorage systems that are connected to each of the primary storagesystems, each of the primary storage systems including a primary logicalvolume that stores the data that is inputted and outputted, each of thesecondary storage systems including: a secondary logical volume thatstores a copy of the data stored in the primary logical volume; and apre-update data storage part that stores pre-update data that is storedbefore data to be stored in the secondary logical volume is updated andtime stamps of the data to be stored in the secondary logical volume, inwhich in each of the secondary storage systems, the pre-update data isstored in the pre-update data storage part, and then write datatransferred from the primary storage system is stored in the secondarylogical volume in an order in which the write data is written into theprimary logical volume; pre-update data prior to a recovery time setbetween a time stamp that is earliest among the latest time stampsstored in the pre-update data storage part and a time stamp that islatest among the earliest time stamps stored in the pre-update datastorage part is obtained from the pre-update data storage part; and thepre-update data is written to the secondary logical volume in an inverseorder of the time stamp with the latest time stamp first, to recoverdata of the secondary logical volume to the recovery time.

Further, in the remote copy system according to the embodiment of thepresent invention, the pre-update data storage part further storespost-update data stored in the secondary logical volume; and in each ofthe secondary storage systems, the time stamp of the latest data that iscurrently stored in the secondary logical volume is compared with theprevious recovery time; when the time stamp of the latest data that iscurrently stored in the secondary logical volume is later than theprevious recovery time, the pre-update data after the recovery time isobtained from the pre-update data storage part and the obtainedpre-update data is written to the secondary logical volume in theinverse order of the time stamp with the latest time stamp first; andwhen the time stamp of the latest data that is currently stored in thesecondary logical volume is earlier than the previous recovery time, thepost-update data prior to the recovery time is obtained from thepre-update data storage part and the obtained post-update data iswritten to the secondary logical volume in the order of the time stampwith the earliest time stamp first.

Further, the remote copy system according to embodiment of the presentinvention further includes a secondary computer that is connected to thesecondary storage system, and in the remote copy system, the secondarycomputer obtains the time stamps stored in the pre-update data storagepart from the secondary storage system, obtains the latest time stampand the earliest time stamp from the time stamps, determines therecovery time between the obtained latest time stamp and the obtainedearliest time stamp, and instructs each of the secondary storage systemsto recover data to the determined recovery time.

Further, in the remote copy system according to embodiment of thepresent invention, the secondary computer obtains the time stamps of thedata stored in the secondary logical volume from the secondary storagesystem, determines, as a deletable time, the earliest time stamp amongthe latest time stamps of the data stored in the secondary logicalvolume, and notifies the secondary storage system of the determineddeletable time, and in the secondary storage system, the pre-update datahaving a time stamp earlier than the received deletable time is deletedfrom the pre-update data storage part.

That is, according to an embodiment of this invention, the primarystorage system transfers a write time appended to a write requesttogether with the data to the secondary storage system. The secondarystorage system obtains pre-update data as a journal before mirroring thetransferred data on the copied data, so that the stored data can berecovered into the pre-update data. Each secondary storage systemmirrors the data received from the primary storage system at anappropriate timing.

When a fault occurs, a management program collects, from each secondarystorage system, information indicating which write time the data hasbeen recovered to, and instructs the secondary storage system to rollback the data to a point in time at which consistency is maintained forthe mirrored data. By executing such a rollback process, the secondarystorage system can recover the consistent data in each secondary storagesystem.

Further, the management program collects, from each secondary storagesystem, information indicating which write time the data has beenmirrored to, and then instructs the secondary storage system to deletethe data to a time at which consistency is maintained for the mirroreddata.

According to embodiment of the present invention, even in the case wherea plurality of primary storage systems and secondary storage systemsexist and operate for asynchronous remote copy, I/O process loads on theprimary storage systems and loads on hosts do not increase. In addition,when a fault occurs, it is possible to maintain consistent data betweena plurality of storage systems. Consequently, this invention iseffectively applied to a large-scale disaster recovery system.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention can be appreciated by the description whichfollows in conjunction with the following figures, wherein:

FIG. 1 is a block diagram of a computer system according to a firstembodiment of this invention.

FIG. 2 is a conceptual diagram of logical volume groups according to thefirst embodiment of this invention.

FIG. 3 is a flow chart of a process performed in the case where astorage system A receives a write request according to the firstembodiment of this invention.

FIG. 4 is a structural diagram of group management information accordingto the first embodiment of this invention.

FIG. 5 is a structural diagram of paired logical volume informationaccording to the first embodiment of this invention.

FIG. 6 is a structural diagram of write data management informationaccording to the first embodiment of this invention.

FIG. 7 is a flow chart of a process of transferring write data from thestorage system A to a storage system B according to the first embodimentof this invention.

FIG. 8 is a flow chart of a process of mirroring write data in thestorage system B according to the first embodiment of this invention.

FIG. 9 is a structural diagram of journal management information.

FIG. 10 is a structural diagram of a journal according to the firstembodiment of this invention.

FIG. 11 is a flow chart of a rollback process for recovering consistencyof the contents of logical volumes in the storage systems B according tothe first embodiment of this invention.

FIG. 12 is a flow chart of a process of deleting journals stored in thestorage system B according to the first embodiment of this invention.

FIG. 13 is a flow chart of a modified example of the process ofmirroring write data in the storage system B according to the firstembodiment of this invention.

FIG. 14 is a structural diagram of a journal in the case of FIG. 13.

FIG. 15 is a flow chart of a rollback/rollforward process for recoveringconsistency of the contents of logical volumes in the storage systems Bin the case of FIG. 13.

FIG. 16 is a block diagram of a computer system according to a secondembodiment of this invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, description will be made of embodiments of this inventionwith reference to the drawings.

FIG. 1 is a block diagram of a computer system according to a firstembodiment of this invention.

The computer system includes a plurality of primary storage systems A100, a plurality of secondary storage systems B 190, a host computer A600, and a host computer B 690. The host computer A 600 and the hostcomputer B 690 are each a computer device including a CPU and a memory.The host computer A 600 is normally used, while the host computer B 690is normally in a standby state.

The primary storage systems A 100 are each connected to the hostcomputer A 600 via an I/O path 900. Also, the secondary storage systemsB 190 are each connected to the host computer B 690 via another I/O path900. The primary storage systems A 100, the secondary storage systems B190, the host computer A 600, and the host computer B 690 are connectedto one another via a network 920.

Further, an operating system (OS) 610 and an application program (APP)620 run on each of the host computer A 600 and the host computer B 690.The application program 620 includes a database management system andother such software. A management program B 890 run on the host computerB 690.

The application program 620 on the host computer A 600 issues an I/Orequest through the operating system 610. The issued I/O request istransferred to the primary storage system A 100 via the I/O path 900.Similarly, the application program 620 on the host computer B 690 issuesan I/O request through the operating system 610. The issued I/O requestis transferred to the secondary storage system B 190 via the I/O path900.

The primary storage systems A 100 each include a control unit 200, acontrol memory 300, a cache 400, and a logical volume 500.

The control unit 200 includes a write data receiving module A 210 and awrite data transferring module A 220. The control unit 200 accesses thecontrol memory 300 and uses information stored in the control memory 300to execute a process described later.

The control memory 300 stores group management information 310, pairedlogical volume information 320, and write data management information330.

The cache 400 is a high speed memory that stores read data and writedata. Each primary storage system A 100 can attain high processingperformance by temporarily storing data in the cache 400.

It should be noted that each unit of the primary storage system A 100 isdesirably made redundant for fault tolerance and usability, and isprovided with a backup power supply.

Similarly, the secondary storage systems B 190 each is provided withanother control unit 200, another control memory 300, another cache 400,and another logical volume 500.

The control unit 200 includes a write data receiving module B 211, awrite data mirroring module B 240, and a journal processing module B260. The control memory 300 and the cache 400 have the same functions asthose of the primary storage system A 100.

The primary storage system A 100 and the secondary storage system B 190provide the logical volumes 500 to the host computer A 600 and the hostcomputer B 690, respectively, as a data storage area. A single logicalvolume 500 is not necessarily structured by a single physical device.For example, the single logical volume 500 may be formed by assemblingstorage areas that are distributed to a plurality of disk drives. Also,the logical volume 500 may have, for example, a mirror structure or aredundant structure such as a RAID structure added with parity data.

The primary storage system A 100 and the secondary storage system B 190are connected to each other via a transfer path 910. As described later,the logical volume of one of the primary storage system A 100 and thesecondary storage system B 190 can store a copy of the content of thelogical volume of the other. According to this embodiment, the copy ofthe content of the logical volume 500 of the primary storage system A100 is stored in the logical volume 500 of the secondary storage systemB 190. In other words, the updated content of the logical volume 500 ofthe primary storage system A 100 is sent to the secondary storage systemB 190 via the transfer path 910. The updated content is then stored inthe logical volume 500 of the secondary storage system B 190.

The data transferred from the primary storage system A 100 to thesecondary storage system B 190 is stored in the logical volume 500 at anarbitrary timing. Accordingly, in some cases, one of two pieces of datathat were simultaneously written to the primary storage systems A 100 ismirrored to the logical volume of one secondary storage system B 190,but the other piece is not mirrored to the logical volume of anothersecondary storage system B 190.

As described later, the primary storage system A 100 and the secondarystorage system B 190 have management information indicating therelationship between their logical volumes 500 in terms of data copy.The management information is used for storing the copied data describedabove in the logical volume 500 of the secondary storage system B 190.The relationship between the logical volumes 500 and the relationshipbetween logical volume groups described later are set by a user asnecessary.

It should be noted that the primary storage systems A 100 and thesecondary storage systems B 190 are not necessarily connected on aone-to-one basis. More specifically, the number of the logical volumesand the number of logical volume groups are not necessarily equal towithin the primary storage systems A 100 and the secondary storagesystems B 190, as long as the source logical volumes and the sourcelogical volume groups correspond to the target logical volumes and thetarget logical volume groups, respectively.

The secondary storage system B 190 stores a journal 700. A storage areafor the journal 700 may be provided separately from the logical volume500, or may be provided in a portion of the storage area of the logicalvolume 500. Alternatively, the journal 700 may be provided not withinthe secondary storage system B 190 but separately from the secondarystorage system B 190.

As shown in FIG. 10, stored in the journal 700 are pre-update data onwhich the write data transferred from the primary storage system A 100to the secondary storage system B 190 is not yet mirrored to the logicalvolume 500, and the management information for the pre-update data. In amodified example shown in FIG. 14, post-update data (write data) to bestored in the logical volume 500 of the secondary storage system B 190is stored in the journal 700 in addition to the pre-update data and themanagement information.

It should be noted that in the above configuration, the host computer B690 is not connected directly to the primary storage systems A 100, butmay be connected to the primary storage systems A 100 via I/O paths (notshown). In this case, when a fault occurs in the host computer A 600 orthe secondary storage system B 190, the host computer B 690 can takeover an operation that has been executed by the host computer A 600 byusing the stored content of the logical volume 500 of the primarystorage system A 100.

Further, in the above configuration, a management program B 890 isexecuted by the host computer B 690, but may be executed by any one ofthe host computer A 600, the primary storage system A 100, and thesecondary storage system B 190. Alternatively, the management program B890 may be executed by another computer (not shown) connected to theprimary storage system A 100 and the secondary storage system B 190.

FIG. 2 is a conceptual diagram of the logical volume group.

In this embodiment, the logical volumes are related to one another on agroup basis. In FIG. 2, broken lines indicate copy relationships betweenthe logical volumes 500 and between the logical volume groups, that is,correspondences between sources and targets. In this embodiment,processes of transferring the write data between the storage systems andmirroring the data in the secondary storage system B 190 are managed insuch a unit of the logical volume group having a plurality of logicalvolumes. Also, resources necessary for the above processes are allocatedto each logical volume group.

If the management of those copy relationships or the allocation of theresources was performed on a logical group basis, a large number ofobjects should be managed and the management would be complicated. Also,many objects to be processed would probably increase the resourcesnecessary for the processes.

On the other hand, if the entire primary storage system A 100 was set asa unit, it would be difficult to perform management corresponding to thecharacteristics of the logical volume 500. In particular, hosts (forexample, main frame hosts and open system hosts) different inperformance required for connection to the logical volume 500 aredivided into separate groups, and the write data is processed in eachgroup. Further, it is preferable that the operation by a user for aprocess of transferring the write data, the setting of tuningconditions, and the like be received on a group basis.

By providing such logical volume groups, it is possible to provide aflexible copy process and the management of the logical volume groupscorresponding to the needs of a user or an operation.

FIG. 3 is a flow chart of a process performed in the case where theprimary storage system A 100 receives from the host computer A 600 awrite request with respect to the source logical volume 500 whose copyis being created. The process is executed by the write data receivingmodule A 210.

The write data receiving module A 210 receives a write request from thehost computer A 600 (step 1000).

The write data receiving module A 210 then stores write data in thecache 400 (step 1001). After that, the group management information 310is referenced to provide the write data with a sequential number andcreate the write data management information 330 (step 1002).

The write data receiving module A 210 finally notifies the host computerA 600 that the write is complete (step 1003).

A write data receiving process described above does not include aprocess of writing the write data stored in the cache 400 to a physicalrecording medium (disk drive) of the logical volume 500, a process oftransferring the write data to the secondary storage system B 190, orother such processes requiring much time. Those processes are executedat appropriate timings after the write data receiving process, that is,asynchronously with the reception of the write data. Therefore, it isonly a short time after the write data receiving module A 210 receivesthe write request until it notifies that the write is complete, therebyrealizing a high speed response with respect to the host computer A 600.

FIG. 4 is a structural diagram of the group management information 310for each logical volume group.

In the group management information 310, information for managing thestructure of the logical volume group is defined. The group managementinformation 310 includes a group ID 311, a sequential number 312, thenumber of logical volumes 313, a logical volume ID 314, a paired storagesystem ID 315, and a paired group ID 316.

The group ID 311 is an identifier that specifies a logical volume groupin the primary storage system A 100.

The sequential number 312 has a value that increments to be sequentiallyprovided to a piece of write data with respect to logical volumesbelonging to the logical volume group. The initial value of thesequential number is, for example, “0”, and the sequential numberincrements by one and is sequentially provided to the piece of writedata.

The number of logical volumes 313 is the number of the logical volumesbelonging to the logical volume group.

The logical volume ID 314 is an identifier that specifies a logicalvolume belonging to the logical volume group uniquely in the primarystorage system A 100.

The paired storage system ID 315 is an identifier that specifies astorage system having a logical volume group paired with the logicalvolume group concerned. In this embodiment, for example, by using aserial number of the secondary storage system B 190 as the pairedstorage system ID 315, a storage system that stores the copy of the datastored in the logical volume belonging to the logical volume groupconcerned is specified.

The paired group ID 316 is a unique identifier that specifies thelogical volume group paired with the logical volume group to which thelogical volume concerned belongs in the paired storage system (secondarystorage system B 190). In other words, the paired group ID 316 specifiesthe logical volume group to which the paired logical volume 500 thatstores the copy of the data stored in the logical volume belonging tothe logical volume group concerned belongs.

FIG. 5 is a structural diagram of the paired logical volume information320 for each logical volume group.

In the paired logical volume information 320, information on a pair ofsource logical volume and target logical volume is defined. The pairedlogical volume information 320 includes a source logical volume ID 321,a target storage system ID 322, and a target logical volume ID 323.

The logical volume ID 321 is an identifier that specifies a sourcelogical volume (logical volume 500 of the primary storage system A 100).

The paired storage system ID 322 is an identifier that specifies asecondary storage system B 190 having a paired logical volume pairedwith the source logical volume specified by the logical volume ID 321.In this embodiment, for example, by using the serial number of thesecondary storage system B 190 as the target storage system ID 322, astorage system that stores the copy of the data stored in the logicalvolume is specified.

The paired logical volume ID 323 is an identifier that specifies thepaired logical volume of the paired storage system B 190. In otherwords, the paired logical volume ID 323 specifies the target logicalvolume 500 that stores the copy of the data stored in the logicalvolume.

FIG. 6 is a structural diagram of the write data management information330 for managing each piece of write data.

In the write data management information 330, information for managingthe write data stored in the logical volume is defined. The write datamanagement information 330 includes a logical volume ID 331, a writeaddress 332, a write data length 333, a write data pointer 334, asequential number 335, a write time 336, and a transfer flag 337.

The logical volume ID 331 is a unique identifier that specifies thelogical volume storing write data.

The write address 332 is a write start address of the write data in thelogical volume.

The write data length 333 is a length of the write data.

The write data pointer 334 is a start address of a storage area for thewrite data in the cache 400.

The sequential number 335 is a numbers that are sequentially provided toa piece of write data in a logical volume group to which a logicalvolume to which the write data is written belongs.

The write time 336 is equal to a time 650 when the host computer A 600issued a write request 630.

The transfer flag 337 is information indicating whether the write dataneeds to be transferred to the secondary storage system B 190. The writedata receiving module A 210 sets the transfer flag 337 when receivingthe write data to create the write data management information 330.

The write data management information 330 is created and managed, forexample, in a list format for each logical volume group.

FIG. 7 is a flow chart of the process of transferring write data fromthe storage system A to the storage system B. Such a transfer process isexecuted by the write data transferring module A 220 of the primarystorage system A 100 and the write data receiving module B 211 of thesecondary storage system B 190.

First, the write data transferring module A 220 references the writedata management information 330 to specify write data to be transferred.After that, the write data transferring module A 220 creates write datainformation relating to the write data to be transferred to thesecondary storage system B 190, by referring to the write datamanagement information 330, group management information 310, and thepaired logical volume information 320 (step 1100).

The write data information created in step 1100 includes the writeaddress 332, the write data length 333, the sequential number 335, andthe write time 336 that are obtained from the write data managementinformation 330. The write data information created above furtherincludes the paired storage system ID 322 and the paired logical volumeID 323 that are obtained from the paired logical volume information 320.The write data information created above further includes the pairedgroup ID 316 obtained from the group management information 310 based onthe logical volume ID 331.

Then, the write data transferring module A 220 transfers the write dataand the write data information created in step 1100 to the secondarystorage system B 190 (step 1101).

After that, the write data receiving module B 211 of the secondarystorage system B 190 stores the received write data and write datainformation in the cache 400 (step 1102). And then the write datareceiving module B 211 creates the write data management information 330from the received write data information (step 1103).

Finally, the write data receiving module B 211 notifies the write datatransferring module A 220 that the write data reception is complete(step 1104). The write data transferring module A 220 of the primarystorage system A 100 which has received such a completion notificationof the write data reception clears the transfer flag of the write datamanagement information 330 into an unset state with respect to the writedata corresponding to the completion notification. At the same time, theprimary storage system A 100 can remove from the cache 400 thetransferred write data that has been held for the transfer to thesecondary storage system B 190.

The write data management information 330 of the secondary storagesystem B 190 includes information on the same items as the write datamanagement information 330 of the primary storage system A 100. The samepieces of data are stored in the write data management information 330of the secondary storage system B 190 except that data for the items aredifferent from those in the write data management information 330 of theprimary storage system A 100.

It should be noted that the logical volume ID 331 is an identifier thatspecifies the target logical volume 500 storing the copy data. The writedata pointer 334 is the start address of a storage area for the piece ofwrite data in the cache 400 of the secondary storage system B 190. Thetransfer flag 337 constantly is not set.

The secondary storage system B 190 holds the group managementinformation 310. The group management information 310 of the secondarystorage system B 190 includes information on the same items as the groupmanagement information 310 of the primary storage system A 100. Itshould be noted that the group ID 311 is an identifier that specifies alogical volume group to which the target logical volume 500 storing thecopy data belongs. The paired storage system ID 315 is an identifierthat specifies the source storage system A 100. The paired group ID 316is an identifier that specifies a logical volume group to which thesource logical volume 500 belongs of the paired storage system A 100.

The secondary storage system B 190 also holds the paired logical volumeinformation 320. The paired logical volume information 320 of thesecondary storage system B 190 includes information on the same items asthe paired logical volume information 320 of the primary storage systemA 100. It should be noted that the logical volume ID 321 is anidentifier that specifies the logical volume 500 storing the copy. Thepaired storage system ID 322 is an identifier that specifies the sourcestorage system A 100. The paired logical volume ID 323 is an ID thatspecifies the source logical volume 500 of the paired storage system A100.

It should be noted that in the process of transferring write datadescribed above, the write data transferring module A 220 firsttransfers write data to the write data receiving module B 211 (step1101). However, the write data receiving module B 211 may issue atransfer request for write data to the write data transferring module A220, and the write data transferring module A 220 that has received thetransfer request may then transfer the write data to the write datareceiving module B 211. By having the transfer request for write dataissued in advance, the timing of transferring write data can becontrolled based on the process conditions, process loads, amounts ofaccumulated write data, etc. in the secondary storage systems B 190.

Also, in the process of transferring write data described above, thewrite data receiving module B 211 stores write data in the cache 400(step 1102). However, a logical volume 500 for storing write data may beprepared aside from the cache 400, and the write data may be stored inthe logical volume 500 for storing write data. In general, the logicalvolume 500 has a larger capacity than the cache 400 and can thereforeaccumulate larger amount of write data.

FIG. 8 is a flow chart of a process of mirroring write data in thesecondary storage system B 190.

Such a mirror process is executed by the write data mirroring module B240, whereby the write data transferred from the primary storage systemA 100 is stored in the logical volume 500 of the secondary storagesystem B 190.

First, the write data mirroring module B 240 selects a piece of writedata management information based on the write time order and thesequential number order, by refereeing to the write data managementinformation 330. According to this step, the order in which pieces ofwrite data are stored in the logical volume 500 is determined (step1200).

After that, the write data mirroring module B 240, by referring to theselected piece of write data management information, identifies thelogical volume ID, the write address, and the write data length. And thewrite data mirroring module B 240 designates a write area of the logicalvolume 500 in which the piece of write data is to be stored (step 1201).

Subsequently, the write data mirroring module B 240 reads data stored inthe write area designated in step 1201 of the logical volume 500 andstores the data in the journal 700 (step 1202). Then, the write time ofthe piece of write data to be stored in the logical volume 500 isregistered as a latest write time 362 of journal management information.Further, the write data mirroring module B 240 updates a journal storageend point 366 based on a location where the journal to which the datahas been added is stored (step 1203).

Finally the write data mirroring module B 240 stores the piece of writedata corresponding to the selected piece of write data managementinformation in the write area specified in step 1201 (step 1204).

According to the above process, the write data transferred from theprimary storage system A 100 is mirrored to the logical volume 500 ofthe secondary storage system B 190. In addition, the past data stored inthe logical volume 500 of the secondary storage system B 190 is storedin the journal 700 to be used in a rollback process (refer to FIG. 11).

FIG. 9 is a structural diagram of journal management information 360provided to each logical volume group.

In the journal management information 360, information for managing thejournal 700 is defined. The journal management information 360 includesa group ID 361, the latest write time 362, an earliest write time 363, adeletable write time 364, a journal storage start point 365, and thejournal storage end point 366.

The group ID 361 is an identifier that specifies the logical volumegroup uniquely.

The latest write time 362 and the earliest write time 363 indicate therange of the write times whose pieces of data are recorded in journals.Specifically, the latest write time 362 indicates the write time whosedata is recorded in the latest journal, and the earliest write time 363indicates the write time whose data is recorded in the earliest journal.

The deletable write time 364 is information indicating the write timewhose data is recorded in the latest journal that can be deleted. Thedeletable write time 364 is used for deleting a journal when the areafor the journal is insufficient.

The journal storage start point 365 and the journal storage end point366 indicate the location where a journal is stored in the logicalvolume.

FIG. 10 is a structural diagram of the journal 700 of the secondarystorage system B 190.

Stored in the journal 700 are data on which the write data transferredfrom the primary storage system A 100 to the secondary storage system B190 is not yet mirrored to the logical volume 500, and the managementinformation for the data. The journal 700 includes a sequential number701, a write time 702, a journal data length 703, a logical volume ID704, a write address 705, a write data length 706, and pre-mirror data707.

The sequential number 701 is an identifier that specifies each piece ofpre-mirror data stored in the journal 700, and is defined to have thevalue increment sequentially by one from “0”.

The write time 702 is the write time of a piece of write data to bemirrored. The journal data length 703 indicates the total length fromthe sequential number 701 to the pre-mirror data 707.

The logical volume ID 704 is an identifier of the logical volume 500that stored the pre-mirror data 707. The write address 705 is an addressof the logical volume 500 where the pre-mirror data 707 is stored. Thewrite data length 706 is the length of the pre-mirror data 707.

The piece of journal data described above includes pre-mirror data 707and the information from the sequential number 701 to the write datalength 706 appended to pre-mirror data 707. The journal 700 containsplural pieces of journal data.

FIG. 11 is a flow chart of a rollback process for recovering consistencyof the contents of logical volumes in the secondary storage systems B190.

First, the management program B 890 requests all the secondary storagesystems B 190 for the latest write time and the earliest write time thatare stored in the journal management information 360 (step 1300).

When each secondary storage system B 190 receives a send request for thelatest write time and the earliest write time, the journal processingmodule B 260 reads the latest write time 362 and the earliest write time363 from the journal management information 360 of each logical volumegroup, and informs the management program B 890 of the range for thejournals stored in the secondary storage system B 190 (step 1301).

After that, the management program B 890 checks whether all thesecondary storage systems B 190 have informed it of the latest writetime and the earliest write time (step 1302).

When the latest write times and the earliest write times are informed byall the secondary storage systems B 190 (step 1303), the managementprogram B 890 obtains a write time that is earliest among the latestwrite times of all the secondary storage systems B 190 (step 1304). Inaddition, the management program B 890 obtains a write time that islatest among the earliest write times of all the secondary storagesystems B 190 (step 1305).

After that, the management program B 890 determines a recovery writetime between the obtained write time that is earliest among the latestwrite times and the obtained write time that is latest among theearliest write times (step 1306). It should be noted that between thewrite time that is earliest among the latest write times obtained andthe write time that is latest among the earliest write times obtained,journals are stored in the logical volumes 500 of all the secondarystorage systems B 190. Therefore, consistency can be maintained byrecovering data of all the secondary storage systems B 190 to any timewithin the range in synchronization with each other. It should be notedthat recovery is normally desired to be made to the latest time that allsystems have consistent data, so that it is desired that the write timethat is earliest among the latest write times and the earliest among thelatest write times obtained in step 1304 is set as the recovery writetime.

The management program B 890 then instructs all the secondary storagesystems B 190 to recover the data to the recovery write time (step1307).

When an instruction to recover the data is received, each secondarystorage system B 190 controls the journal processing module B 260, byreferring to the journal 700 writes pieces of pre-mirror data 707 to thelogical volume 500 in the inverse order of the write time 702 (thelatest write time first) to recover the content of the logical volume500 to the recovery write time (step 1308).

When the logical volume 500 is recovered, the journal processing moduleB 260 notifies the management program B 890 that the recovery iscomplete (step 1309).

After that, the management program B 890 checks whether or not all thesecondary storage systems B 190 have notified it that the recovery iscomplete (step 1310).

When the recovery is complete in all the secondary storage systems B 190(step 1311), the management program B 890 informs the operating system610 and the application program 620 of the host computer B 690 that thelogical volumes 500 of the secondary storage systems B 190 are usable(step 1312).

As described above, in the rollback process, the recovery write time isdetermined between the obtained write time that is earliest among thelatest write times and the obtained write time that is latest among theearliest write times. All the secondary storage systems B 190 have thedata recovered to the recovery write time in synchronization with eachother.

FIG. 12 is a flow chart of a process of deleting journals stored in thesecondary storage system B 190.

First, the management program B 890 requests all the secondary storagesystems B 190 for the latest write time stored in the journal 700 (step1400).

When each secondary storage system B 190 receives a send request for thelatest write time, the journal processing module B 260 reads the latestwrite time 362 from the journal management information 360 of eachlogical volume group, and informs the management program B 890 of thelatest write time for the journals stored in the secondary storagesystem B 190 (step 1401).

After that, the management program B 890 checks whether all thesecondary storage systems B 190 have informed it of the latest writetime (step 1402).

When the latest write times are informed by all the secondary storagesystems B 190 (step 1403), the management program B 890 obtains a writetime that is earliest among the latest write times of all the secondarystorage systems B 190. Then, the management program B 890 determines theobtained write time that is earliest among the latest write times as adeletable write time (step 1404). It should be noted that, the writedata corresponding the journals up to the obtained write time that isearliest among the latest write times has been mirrored in the logicalvolumes 500 of all the secondary storage systems B 190. Therefore, thejournal 700 for pieces of data of up to the obtained write time that isearliest among the latest write times is no longer used for the rollbackprocess (FIG. 11).

The management program B 890 then informs all the secondary storagesystems B 190 of the determined deletable write time (step 1405).

When the deletable write time is received, each secondary storage systemB 190 controls the journal processing module B 260 to record thereceived deletable write time as the deletable write time 364 of thejournal management information (step 1406).

In the case where the amount of data of journals increases to make apredetermined journal storage area insufficient, the journal processingmodule B 260 of the secondary storage system B 190, by referring to thejournal management information 360, deletes all or part of the journalswithin the write time range between the earliest write time and thedeletable write time. Accordingly, the latest write time among the writetimes for stored journals is registered as the latest write time 362 ofthe journal management information. Further, the journal storage startpoint 365 and/or the journal storage end point 366 are updated based onthe location where the deleted journal was stored (step 1407).

The process of deleting journals described above is executed in thebackground of normal processes at predetermined timings (e.g.,periodically). Further, the secondary storage system B 190 may monitorthe data amount of journals, and the journal processing module B 260 mayrequest the management program B 890 to execute the process of deletinga journal when the data amount is larger than a predetermined value.

As described above, in the process of deleting a journal, the obtainedwrite time that is earliest among the latest write times is determinedas the deletable write time. The journal data whose write time is equalto or earlier than the deletable write time is not required for the datarecovery of the secondary storage system B 190, so that the secondarystorage system B 190 deletes the journal data as necessary.

FIG. 13 is a flow chart of the modified example of the process ofmirroring write data in the secondary storage system B 190.

The modified example of the process of mirroring write data is differentfrom the process of mirroring write data described above (FIG. 8) inthat the write data itself is also stored in a journal.

First, the write data mirroring module B 240 selects a piece of writedata management information based on the write time order and thesequential number order, by referring to the write data managementinformation 330. According to this step, the order in which pieces ofwrite data are stored in the logical volume 500 is determined (step1500).

After that, the write data mirroring module B 240, by referring to theselected piece of write data management information, identifies thelogical volume ID, the write address, and the write data length. And thewrite data mirroring module B 240 designates the write area of thelogical volume 500 in which the piece of write data is to be stored(step 1501).

Subsequently, the write data mirroring module B 240 reads data stored inthe write area designated in step 1501 of the logical volume 500 andstores the read out data and the piece of write data in the journal 700(step 1502). Then, the write data mirroring module B 240 updates thelatest write time 361 and the journal storage end point 366 of thejournal management information (step 1503).

Finally the write data mirroring module B 240 stores the piece of writedata corresponding to the selected piece of write data managementinformation in the write area specified in step 1501 (step 1504).

According to the above process, the write data transferred from theprimary storage system A 100 is mirrored to the logical volume 500 ofthe secondary storage system B 190. In addition, the past data stored inthe logical volume 500 of the secondary storage system B 190 and thedata to be written to the logical volume 500 are stored in the journal700 to be used in a rollback process (refer to FIG. 15).

FIG. 14 is a structural diagram of the journal 700 according to themodified example of the write data mirroring process (FIG. 13).

Stored in the journal 700 are data on which the write data transferredfrom the primary storage system A 100 to the secondary storage system B190 is not yet mirrored to the logical volume 500, write data to bestored in the logical volume 500 of the secondary storage system B 190,and the management information for the data.

The journal according to the modified example includes a sequentialnumber 701, a write time 702, a journal data length 703, a logicalvolume ID 704, a write address 705, a write data length 706, pre-mirrordata 707, and post-mirror data (write data) 708. It should be noted thatpieces of data other than the post-mirror data 708 are the same as thoseof the journal (FIG. 10), and their detailed description will beomitted.

Further, the sequential number 701, the write time 702, and the journaldata length 703 are stored on a significant address side of the jounal.The information consisting of 701, 702, and 703 stored on thesignificant address side of the post-mirror data 708 are used forsearching the journal 700 from the significant address. It should benoted that, in the case where the journal 700 is searched from thesignificant address, the first portion (including the write time 702 andthe journal data length 703) of a unit journal has only to be read todetermine whether the unit journal is necessary for recovering process,so that the data of the next journal can be read with efficiency.

FIG. 15 is a flow chart of a rollback/rollforward process for recoveringconsistency of the contents of logical volumes 500 in the secondarystorage systems B 190 according to the modified example of the writedata mirroring process (FIG. 13).

First, the management program B 890 requests all the secondary storagesystems B 190 for the latest write time and the earliest write time thatare stored in the journal 700 (step 1600).

When each secondary storage system B 190 receives a send request for thelatest write time and the earliest write time, the journal processingmodule B 260 reads the latest write time 362 and the earliest write time363 from the journal management information 360 of each logical volumegroup, and informs the management program B 890 of the range for thejournals stored in the secondary storage system B 190 (step 1601).

After that, the management program B 890 checks whether all thesecondary storage systems B 190 have informed it of the latest writetime and the earliest write time (step 1602).

When the latest write times and the earliest write times are informed byall the secondary storage systems B 190 (step 1603), the managementprogram B 890 obtains the write time that is earliest among the latestwrite times of all the secondary storage systems B 190 (step 1604). Inaddition, the management program B 890 obtains the write time that islatest among the earliest write times of all the secondary storagesystems B 190 (step 1605).

After that, the management program B 890 determines the recovery writetime between the write time that is earliest among the latest writetimes obtained and the write time that is latest among the earliestwrite times obtained (step 1606). It should be noted that between thewrite time that is earliest among the latest write times obtained andthe write time that is latest among the earliest write times obtained,journals are stored in the logical volumes 500 of all the secondarystorage systems B 190. Therefore, data of all the secondary storagesystems B 190 can be recovered to any time within the range insynchronization with each other. It should be noted that recovery isnormally desired to be made to the latest time that all systems haveconsistent data, so that it is desired that the write time that isearliest among the latest write times and the earliest among the latestwrite times obtained in step 1604 is set as the recovery write time.

The management program B 890 then instructs all the secondary storagesystems B 190 to recover the data to the recovery write time (step1607).

When an instruction to recover the data are received, each secondarystorage system B 190 controls the journal processing module B 260, byreferring to the journal management information 360, judges whether thedetermined recovery write time is earlier or later than a previouslyrecovered write time (step 1608). The previously recovered write time isrecorded prior to the rollback/rollforward process (FIG. 15) as thewrite time of data recovered when another rollback process (FIG. 11) oranother rollback/rollforward process (FIG. 15) is executed.

When a result from the judgment indicates that the determined recoverywrite time is earlier than the previously recovered write time, thejournal processing module B 260, by referring to the journal 700, writespieces of pre-mirror data 707 to the logical volume 500 in the inverseorder of the write time 702 (the latest write time first) to recover thecontent of the logical volume 500 to the recovery write time (step1609). After that, the process goes to step 1611.

On the other hand, when the determined recovery write time is later thanthe previously recovered write time, the journal processing module B260, by referring to the journal 700, writes pieces of post-mirror data708 to the logical volume 500 in the order of the write time 702 (theearliest write time first) to mirror the content of the logical volume500 to the recovery write time (step 1610). After that, the processadvances to step 1611.

When the logical volume 500 is recovered, the journal processing moduleB 260 informs the management program B 890 that the recovery is complete(step 1611).

After that, the management program B 890 checks whether all thesecondary storage systems B 190 have informed it that the recovery iscomplete (step 1612).

When the recovery is complete in all the secondary storage systems B 190(step 1613), the management program B 890 informs the operating system610 and the application program 620 of the host computer B 690 that thelogical volumes 500 of the secondary storage systems B 190 are usable(step 1614).

As described above, in the modified example of the rollback process,when the recovery write time is earlier than the previously recoveredwrite time, the pre-mirror data is used to recover the data to therecovery write time. When the recovery write time is later than thepreviously recovered write time, the post-mirror data is used to recoverthe data to the recovery write time. Thus, the data can be recovered toany write time.

In the various processes described above, various instructions,notifications, and pieces of information that are sent/received betweenthe primary storage system A 100 or the secondary storage system B 190and the management program B 890 may pass either the I/O path 900 or thenetwork 920.

FIG. 16 is a block diagram of a computer system according to a secondembodiment of this invention.

The computer system according to the second embodiment is different fromthe computer system according to the first embodiment in that the hostcomputer A 600 is connected to a plurality of primary storage systems C180 through the I/O paths 900, and the plurality of primary storagesystems C 180 are connected to each of the plurality of local storagesystems A 100 through the transfer paths 910.

More specifically, the computer system according to the secondembodiment is provided with three-stage storage systems consisting ofthe primary storage systems C 180, the local storage systems A 100, andthe remote storage systems B 190. The local storage systems A 100 areset relatively near to (for example, several km from) the primarystorage systems C 180. The remote storage systems B 190 are set remotely(for example, several hundreds of km) from the local storage systems A100. The respective storage systems are connected in series (cascadeconnection) in the order of the primary storage systems C 180, the localstorage system A 100, and the remote storage system B 190.

In the second embodiment, the copy of the data stored in the logicalvolumes 500 of the primary storage systems C 180 is stored in thelogical volumes 500 of the local storage systems A 100. This process forcopying data from the primary storage systems C 180 to the local storagesystems A 100 is executed as a synchronous copy process.

Further, the copy of the data stored in the logical volumes 500 of thelocal storage systems A 100 is stored in the logical volumes 500 of theremote storage systems B 190. This process for copying data from thelocal storage systems A 100 to the remote storage systems B 190 isexecuted as the same asynchronous copy process as described in the firstembodiment.

In other words, in the second embodiment, the copy of the data stored inthe logical volumes 500 of the primary storage systems C 180 is storedin the local storage systems A 100 and in the remote storage systems B190. In order to attain this process, the primary storage systems C 180are provided with the same configurations, various pieces ofinformation, etc. as the local storage systems A 100 described in thefirst embodiment.

It should be noted that the local storage systems A 100 and the remotestorage systems B 190 of the second embodiment have the sameconfigurations and functions as the local storage systems A 100 and theremote storage systems B 190 of the first embodiment, respectively.Thus, their detailed description will be omitted here.

When a write request to write data to the logical volume 500 is receivedfrom the host computer A 600, the primary storage system C 180 storesreceived write data to the logical volume 500 of the primary storagesystem C 180. Further, a write data transferring module C 222 transfersthe received write data and write time to the write data receivingmodule A 210 of the local storage system A 100.

At this time, as described above, the process of transferring data fromthe primary storage system C 180 to the local storage system A 100 isexecuted as a synchronous backup process. Specifically, after theprimary storage system C 180 is informed by the write data receivingmodule A 210 that it has received the data, the primary storage system C180 informs the host computer A 600 that the write is complete.Accordingly, the primary storage system C 180 ensures that the copy ofthe write data for which the write request has been issued exists in thelocal storage system A 100.

Therefore, for example, in the case where a fault occurs in the localstorage system A 100 or the transfer path 910 and data cannot betransferred to the local storage system A 100, the host computer A 600recognizes that the write process is not complete on the write data thathas not been transferred to the local storage system A 100. Also, thewrite data received by the primary storage system C 180 is reliablystored in the local storage system A 100 as well, so that the expectedcopy is stored in the local storage system A 100. Further, after all thewrite data received by the local storage system A 100 is transferred tothe remote storage system B 190, the expected copy is stored also in theremote storage system B 190. Accordingly, when the process beingexecuted by the host computer A 600 is interrupted, the host computer B690 can take over an operation by using the same data as the datarecognized by the host computer A 600 as having been written.

Further, by the same process as the first embodiment, the copy of thedata stored in the logical volume 500 of the local storage system A 100is stored in the logical volume 500 of the remote storage system B 190.According to this process, even in the case where, for example, a faultoccurs in the local storage system A 100 or the transfer path 910 anddata cannot be transferred to the local storage system A 100, when thehost computer A 600 interrupts the process, the host computer B 690 canobtain the same data as the data stored in the primary storage system C180 from the remote storage system B 190, and can therefore take over anoperation by using the data.

While the present invention has been described in detail and pictoriallyin the accompanying drawings, the present invention is not limited tosuch detail but covers various obvious modifications and equivalentarrangements, which fall within the purview of the appended claims.

1. A remote copy system for copying data between a plurality of storagesystems, comprising: a plurality of first storage systems to and fromwhich data is inputted and outputted, and a plurality of second storagesystems that are connected to each of the first storage systems; each ofthe first storage systems including a first logical volume that storesthe data that is inputted and outputted; each of the second storagesystems including a second logical volume that stores a copy of the datastored in the first logical volume; the remote copy system comprising apre-update data storage unit that stores pre-update data that is storedbefore data to be stored in the second logical volume is updated andtime stamps of the data to be stored in the second logical volume,wherein in each of the second storage systems, the pre-update data isstored in the pre-update data storage unit, and then write datatransferred from the first storage system is stored in the secondlogical volume in an order in which the write data is written into thefirst logical volume; pre-update data prior to a recovery time setbetween a time stamp that is earliest among each of latest time stampsof the second logical volumes stored in the pre-update data storage unitand a time stamp that is latest among each of earliest time stamps ofthe second logical volumes stored in the pre-update data storage unit isobtained from the pre-update data storage unit; and the pre-update datais written to each of the second logical volumes in an inverse order ofthe time stamp with a latest time stamp first, to recover data of eachof the second logical volumes to the recovery time.
 2. The remote copysystem according to claim 1, wherein the time stamp includes a time atwhich data is inputted to the first storage system.
 3. The remote copysystem according to claim 1, wherein the recovery time includes anearliest time stamp among the latest time stamps of the data stored inthe second logical volume.
 4. The remote copy system according to claim1, wherein: the pre-update data storage unit further stores post-updatedata stored in the second logical volume; and in each of the secondstorage systems, the time stamp of the latest data that is currentlystored in the second logical volume is compared with the recovery time;when the time stamp of the latest data that is currently stored in thesecond logical volume is later than the recovery time, the pre-updatedata after the recovery time is obtained from the pre-update datastorage unit and the obtained pre-update data is written to the secondlogical volume in the inverse order of the time stamp with the latesttime stamp first; and when the time stamp of the latest data that iscurrently stored in the second logical volume is earlier than therecovery time, the post-update data prior to the recovery time isobtained from the pre-update data storage unit and the obtainedpost-update data is written to the second logical volume in the order ofthe time stamp with the earliest time stamp first.
 5. The remote copysystem according to claim 1, further comprising: a first computer thatis connected to the first storage system; and a second computer that isconnected to the second storage system, wherein: each of the secondstorage systems comprises the pre-update data storage unit; the firstcomputer inputs and outputs data to and from the first storage system;and the second computer: obtains the time stamps stored in thepre-update data storage unit from the second storage system; obtains thelatest time stamp and the earliest time stamp from the obtained timestamps; determines the recovery time between the obtained latest timestamp and the obtained earliest time stamp; and instructs each of thesecond storage systems to recover data to the determined recovery time.6. The remote copy system according to claim 5, wherein: the secondcomputer: obtains the time stamps of the data stored in the secondlogical volume from the second storage system; determines, as adeletable time, the earliest time stamp among the latest time stamps ofthe data stored in the second logical volume; and informs the secondstorage system of the determined deletable time; and in the secondstorage system, the pre-update data having a time stamp earlier than thereceived deletable time is deleted from the pre-update data storageunit.
 7. A remote copy system for copying data between a plurality ofstorage systems, comprising: a plurality of first storage systems to andfrom which data is inputted and outputted, a plurality of second storagesystems that are connected to each of the first storage systems, and aplurality of third storage systems that are connected to each of thesecond storage systems, each of the first storage systems including afirst logical volume that stores the data that is inputted andoutputted; each of the second storage systems including a second logicalvolume that stores a copy of the data stored in the first logicalvolume; each of the third storage systems comprising a third logicalvolume that stores a copy of the data stored in the second logicalvolume; the remote copy system comprising a pre-update data storage unitthat stores pre-update data that is stored before data to be stored inthe third logical volume is updated and time stamps of the data to bestored in the third logical volume, wherein in each of the third storagesystems, the pre-update data is stored in the pre-update data storageunit, and then write data transferred from the second storage system isstored in the third logical volume in an order in which the write datais written into the first logical volume; pre-update data prior to arecovery time set between a time stamp that is earliest among each oflatesttime stamps of the third loqical volumes stored in the pre-updatedata storage unit and a time stamp that is latest among each of earliesttime stamps of the third loqical volumes stored in the pre-update datastorage unit is obtained from the pre-update data storage unit; and thepre-update data is written to each of the third logical volumes in aninverse order of the time stamp with a latest time stamp first, torecover data of each of the third logical volume to the recovery time.8. The remote copy system according to claim 7, wherein the time stampincludes a time at which data is inputted to the first storage system.9. The remote copy system according to claim 7, wherein the recoverytime includes an earliest time stamp among the latest time stamps of thedata stored in the second logical volume.
 10. The remote copy systemaccording to claim 7, wherein: the pre-update data storage unit furtherstores post-update data stored in the third logical volume; and in eachof the third storage systems, the time stamp of the latest data that iscurrently stored in the third logical volume is compared with theprevious recovery time; when the time stamp of the latest data that iscurrently stored in the third logical volume is later than the recoverytime, the pre-update data after the recovery time is obtained from thepre-update data storage unit and the obtained pre-update data is writtento the third logical volume in the inverse order of the time stamp withthe latest time stamp first; and when the time stamp of the latest datathat is currently stored in the third logical volume is earlier than therecovery time, the post-update data prior to the recovery time isobtained from the pre-update data storage unit and the obtainedpost-update data is written to the third logical volume in the order ofthe time stamp with the earliest time stamp first.
 11. The remote copysystem according to claim 7, further comprising: a first computer thatis connected to the first storage system; and a second computer that isconnected to the third storage system, wherein: each of the secondstorage systems comprises the pre-update data storage unit; the firstcomputer inputs and outputs data to and from the first storage system;and the second computer: obtains the time stamps stored in thepre-update data storage unit from the second storage system; obtains thelatest time stamp and the earliest time stamp from the obtained timestamps; determines the recovery time between the obtained latest timestamp and the obtained earliest time stamp; and instructs each of thethird storage systems to recover data to the determined recovery time.12. The remote copy system according to claim 11, wherein: the secondcomputer: obtains the time stamps of the data stored in the thirdlogical volume from the third storage system; determines, as a deletabletime, the earliest time stamp among the latest time stamps of the datastored in the third logical volume; and informs the third storage systemof the determined deletable time; and in the third storage system, thepre-update data having a time stamp earlier than the received deletabletime is deleted from the pre-update data storage unit.
 13. A storagesystem for a plurality of secondary storage systems that are connectedto each of a plurality of primary storage systems and copy data storedin the plurality of primary storage systems; each of the primary storagesystems comprising a primary logical volume that stores data that isinputted and outputted from a computer; each of the secondary storagesystems including a secondary logical volume that stores a copy of thedata stored in the primary logical volume, and a pre-update data storageunit that stores pre-update data that is stored before data to be storedin the secondary logical volume is updated and time stamps of the datato be stored in the secondary logical volume; wherein: the pre-updatedata is stored in the pre-update data storage unit, and then write datatransferred from the primary storage system is stored in the secondarylogical volume in an order in which the write data is written into theprimary logical volume; pre-update data prior to a recovery time setbetween a time stamp that is earliest among each of latest time stampsof the secondary logical volumes stored in the pre-update data storageunit and a time stamp that is latest among each of earliest time stampsof the secondary logical volumes stored in the pre-update data storageunit is obtained from the pre-update data storage unit; and thepre-update data is written to each of the secondary logical volumes inan inverse order of the time stamp with a latest time stamp first, torecover data of each of the secondary logical volume to the recoverytime.
 14. The storage system according to claim 13, wherein the timestamp includes a time at which data is inputted to the primary storagesystem.
 15. The storage system according to claim 13, wherein therecovery time incIudes an earliest time stamp among the latest timestamps of the data stored in the secondary logical volume.
 16. Thestorage system according to claim 13, wherein: the pre-update datastorage unit further stores post-update data stored in the secondarylogical volume; the time stamp of the latest data that is currentlystored in the secondary logical volume is compared with the recoverytime; when the time stamp of the latest data that is currently stored inthe secondary logical volume is later than the recovery time, thepre-update data after the recovery time is obtained from the pre-updatedata storage unit and the obtained pre-update data is written to thesecondary logical volume in the inverse order of the time stamp with thelatest time stamp first; and when the time stamp of the latest data thatis currently stored in the secondary logical volume is earlier than therecovery time, the post-update data prior to the recovery time isobtained from the pre-update data storage unit and the obtainedpost-update data is written to the secondary logical volume in the orderof the time stamp with the earliest time stamp first.
 17. The storagesystem according to claim 13, further: obtains the time stamps stored inthe pre-update data storage unit from each of the secondary storagesystem, obtains the latest time stamp and the earliest time stamp fromthe obtained time stamps, determines the recovery time between theobtained latest time stamp and the obtained earliest time stamp, andrecovers data to the determined recovery time.
 18. The storage systemaccording to claim 17, further: obtains the time stamps of the datastored in the secondary logical volume from the secondary storagesystem, determines, as a deletable time, the earliest time stamp amongthe latest time stamps of the data stored in the secondary logicalvolume, informs another secondary storage system of the determineddeletable time, and deletes the pre-update data having a time stampearlier than the determined deletable time from the pre-update datastorage unit.